Automated segmentation and guided correction of endothelial cell images

ABSTRACT

Embodiments discussed herein facilitate automated and/or semi-automated segmentation of endothelial cells via a trained deep learning model. One example embodiment is a method, comprising: accessing an optical microscopy image comprising a set of corneal endothelial cells of a patient of a keratoplasty; pre-processing the optical microscopy image to generate a pre-processed optical microscopy image via correcting for at least one of shading or illumination artifacts in the optical microscopy image; segmenting, based at least in part on a trained deep learning (DL) model, a plurality of corneal endothelial cells of the set of corneal endothelial cells in the pre-processed optical microscopy image; and displaying, via a graphical user interface (GUI), at least the segmented plurality of corneal endothelial cells.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/985,481 filed Mar. 5, 2020, entitled “AUTOMATEDSEGMENTATION AND GUIDED CORRECTION OF ENDOTHELIAL CELL IMAGES”, thecontents of which are herein incorporated by reference in theirentirety.

FEDERAL FUNDING NOTICE

This invention was made with government support under the grant(s)EY029498 awarded by the National Institutes of Health. The governmenthas certain rights in the invention.

BACKGROUND

The Eye Bank Association of America (EBAA) reported 48,366, cornealkeratoplasties performed in the US in 2017, with 28,993 endothelialkeratoplasty (EK) and 18,346 penetrating keratoplasty (PK) procedures.The regraft rate for 2016 was 8.8% for EK and 11.8% for PK, where graftfailure involves regrafting for any reason or a cornea which remainscloudy without clearing for at least 90 days of observation.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various example operations,apparatus, methods, and other example embodiments of various aspectsdiscussed herein. It will be appreciated that the illustrated elementboundaries (e.g., boxes, groups of boxes, or other shapes) in thefigures represent one example of the boundaries. One of ordinary skillin the art will appreciate that, in some examples, one element can bedesigned as multiple elements or that multiple elements can be designedas one element. In some examples, an element shown as an internalcomponent of another element may be implemented as an external componentand vice versa. Furthermore, elements may not be drawn to scale.

FIG. 1 illustrates a flow diagram of an example method/set of operationsthat can be performed by one or more processors to segment cornealendothelial cells in an automatic and/or semi-automatic manner via atrained deep learning model, according to various aspects discussedherein.

FIG. 2 illustrates a flow diagram of an example method/set of operationsthat can be performed by one or more processors to train a model viadeep learning to segment endothelial cells, according to various aspectsdiscussed herein.

FIG. 3 illustrates a flow diagram of an example method/set of operationsthat can be performed by one or more processors to determine a prognosisfor a keratoplasty based on features calculated from segmented cornealendothelial cells, according to various aspects discussed herein.

FIG. 4 illustrates a time comparison overview of the manual andautomated techniques employed to segment an initial and raw endothelialcell image along with the flow diagram of the implemented automaticsegmentation algorithm, in connection with various aspects discussedherein.

FIG. 5 illustrates serial specular microscope imaging following apenetrating keratoplasty (PK) procedure showing a continuing decrease inendothelial cell density, in connection with various aspects discussedherein.

FIG. 6 illustrates example images showing four types of graph structuresthat can be employed to capture cell arrangement, in connection withvarious embodiments discussed herein.

FIG. 7 illustrates example images showing Delauney graph analysis of twogroups (top and bottom rows) with clear differences of centrallylocated, repeated EC images, in connection with various aspectsdiscussed herein.

FIG. 8 illustrates a graph showing statistically different features(those above the 0.95 line) as computed from two sets of repeated,central EC images as in FIG. 7, in connection with various aspectsdiscussed herein.

FIG. 9 illustrates images showing image processing to identify darknuclei and bright pigments in EC images, in connection with variousaspects discussed herein.

FIG. 10 illustrates a graph and heatmap of an example EC image showingvariations in thickness of dark gaps between cells, measured via thefull width at half-intensity-minimum of the curve (left), in connectionwith various aspects discussed herein.

FIG. 11 illustrates example images associated with a post-processingpipeline of the example use cases, in connection with various aspectsdiscussed herein.

FIG. 12 illustrates example images showing hexagonality determined basedon shape detector techniques of the example use cases, in connectionwith various aspects discussed herein.

FIG. 13 illustrates a diagram showing an overview of training, testing,and application of the deep learning model for endothelial cellsegmentation, in connection with various aspects discussed herein.

FIG. 14 illustrates specular microscopy images with varying endothelialcell density, duration post-transplant, and contrast from left to rightacross the image area, in connection with various aspects discussedherein.

FIG. 15 illustrates a diagram showing the U-Net architecture that can beemployed in connection with various aspects discussed herein.

FIG. 16 illustrates images showing two different thresholding techniquesthat can be employed to convert probability outputs to binarizedoutputs, followed by a post-processing pipeline consisting of fourmorphological operations, in connection with various embodiments.

FIG. 17 illustrates images showing an example endothelial cell image,manual cell segmentation analysis, and the corresponding ground truth,corrected for border overlaps, used in the example use cases, inconnection with various aspects discussed herein.

FIG. 18 illustrates example images showing three pre-processing shadingcorrection methods applied to a representative EC image, in connectionwith various aspects discussed herein.

FIG. 19A illustrates images of deep learning probability maps of anexample test image following four types of pre-processing, in connectionwith various aspects discussed herein.

FIG. 19B illustrates images showing the final segmentation results ofthe EC image of FIG. 20 with the four pre-processing methods discussedabove, according to various aspects discussed herein.

FIG. 20A illustrates three example EC images and correspondingsegmentation highlighting cells with differences between manual andautomatic segmentations, in connection with various aspects discussedherein.

FIG. 20B illustrates three examples of EC images showing the automaticsegmentation method identifying and segmenting cells outside of thesample area of the ground truth annotations, in connection with variousaspects discussed herein.

FIG. 21 illustrates charts showing the results of visual analysis ofautomated cells in a held-out test set comprising 30 images, inconnection with various aspects discussed herein.

FIG. 22 illustrates images showing three example post-DSAEK EC images(corresponding to the three rows) having undergone both manual andautomatic segmentation, in connection with various aspects discussedherein.

FIG. 23 illustrates an example screenshot of semi-automated segmentationgraphic user interface (GUI) software, in connection with variousaspects discussed herein.

FIG. 24 illustrates example images showing cells with outlier areas orcells that were under-segmented and/or had faint or incomplete bordersnoticeable in the corresponding cell's area of the probability output,in connection with various aspects discussed herein.

FIG. 25 illustrates Bland Altman plots for endothelial cell density(ECD), coefficient of variation (CV) of cell area, and hexagonality(HEX), in connection with various aspects discussed herein.

FIG. 26 illustrates a heatmap of an example EC image, showing borderthickness, in connection with various aspects discussed herein.

FIG. 27 illustrates a confusion matrix showing predicted and actualclassification results for a machine learning classifier trained andtested on a small cohort of EC images from the Netherlands Institute ofInnovative Ocular Surgery, in connection with various aspects discussedherein.

FIG. 28 illustrates five examples of clinical quality EC images taken atvarious time points following DSAEKs with this illumination gradient, inconnection with various aspects discussed herein.

FIG. 29 illustrates a diagram of an example apparatus that canfacilitate training and/or employing deep learning model(s) toautomatically and/or semi-automatically segment endothelial cells,according to various embodiments discussed herein.

DETAILED DESCRIPTION

In light of the risks of graft failure via rejection or otherwise,successful predictive image analytics of potential failure can beadvantageous. Via increased surveillance, patients can be identified fortailoring topical corticosteroid usage associated with some glaucomarisk, and counseling can be intensified to improve patient compliance.Compliance with medication is an issue with corneal grafts, and it islikely that interventions could improve results in a manner similar tothe well-studied effect in glaucoma. Saving the initial graft is quiteimportant, as there is a significant increase in rejection rate in asecond graft, with mechanisms identified in mouse studies. Byidentifying possible rejection at a subclinical level, this couldpotentially reduce graft rejection failures significantly. Variousembodiments can facilitate the identification of potential rejectionand/or graft failure, which can reduce health care costs, patientdiscomfort, patient angst, and vision loss.

Evidence suggests that corneal EC images could predict graft rejectionand failure. The corneal endothelium plays a critical role inmaintaining stromal clarity. A sufficient number of endothelial cells,serving as Na+/K+-ATPase pump sites, are required to prevent cornealswelling which impairs vision and can ultimately lead to graft failure.The normal endothelial cell layer has a regular hexagonal structure(nature's preferred cellular arrangement, as described below). Theliterature suggests that changes in morphometric measures, includingcoefficient of variation (CV) of cell area (polymegathism) and thepercentage of hexagonal cells or hexagonality (HEX), reflectingvariation in cell shape (pleomorphism), may be more sensitive thanendothelial cell density (ECD) in assessing endothelial health anddysfunction. Research performed in connection with various embodimentsbased on the National Eye Institute-sponsored Specular MicroscopyAncillary Study (SMAS) images found that 6-month HEX results weresuggestive of an association with subsequent late graft failure, whereasCV was not predictive of graft failure. Consultants, Drs. Baydoun andMelles of the Netherlands Institute for Innovative Ocular Surgery(NIIOS), created a scoring system predictive of corneal rejection afterDescemet Membrane Endothelial Keratoplasty (DMEK) surgery. In additionto CV and HEX, they described visual subjective scoring related to thecell morphology pattern and distribution, cellular reflectivity,presence/size of cell nuclei and appearance of cellular activation.Scoring metrics were determined by comparing rejection and non-rejectionimage groups, a process systematized in connection with variousembodiments and the example use cases discussed below.

Various embodiments discussed herein can comprise and/or employtechniques that can facilitate segmenting corneal endothelial cellsand/or training a deep learning model to perform such segmentation.Various embodiments can be applied to corneal endothelial cells imagesacquired with various types of specular and confocal microscopes.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a memory. These algorithmic descriptions and representationsare used by those skilled in the art to convey the substance of theirwork to others. An algorithm, here and generally, is conceived to be asequence of operations that produce a result. The operations may includephysical manipulations of physical quantities. Usually, though notnecessarily, the physical quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a logic or circuit, and so on.The physical manipulations create a concrete, tangible, useful,real-world result.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, and so on. It should be borne in mind,however, that these and similar terms are to be associated with theappropriate physical quantities and are merely convenient labels appliedto these quantities. Unless specifically stated otherwise, it isappreciated that throughout the description, terms including processing,computing, calculating, determining, and so on, refer to actions andprocesses of a computer system, logic, circuit, processor, or similarelectronic device that manipulates and transforms data represented asphysical (electronic) quantities.

Example methods and operations may be better appreciated with referenceto flow diagrams. While for purposes of simplicity of explanation, theillustrated methodologies are shown and described as a series of blocks,it is to be appreciated that the methodologies are not limited by theorder of the blocks, as some blocks can occur in different orders and/orconcurrently with other blocks from that shown and described. Moreover,less than all the illustrated blocks may be required to implement anexample methodology. Blocks may be combined or separated into multiplecomponents. Furthermore, additional and/or alternative methodologies canemploy additional, not illustrated blocks.

Referring to FIG. 1, illustrated is a flow diagram of an examplemethod/set of operations 100 that can be performed by one or moreprocessors to segment corneal endothelial cells in an automatic and/orsemi-automatic manner via a trained deep learning model, according tovarious aspects discussed herein.

The set of operations 100 can comprise, at 110, accessing an opticalmicroscopy image comprising a set of corneal endothelial cells of apatient of a keratoplasty. In various embodiments and in the example usecases discussed below, the image can be obtained via a system and/orapparatus implementing the set of operations 100, or can be obtainedfrom a separate medical imaging system. Additionally, the training setof images can be accessed contemporaneously with or at any point priorto performing the set of operations 100.

The set of operations 100 can further comprise, at 120, pre-processingthe optical microscopy image to generate a pre-processed opticalmicroscopy image via correcting for at least one of shading orillumination artifacts in the optical microscopy image;

The set of operations 100 can further comprise, at 130, segmenting,based at least in part on a trained deep learning (DL) model, aplurality of corneal endothelial cells of the set of corneal endothelialcells in the pre-processed optical microscopy image.

The set of operations 100 can further comprise, at 140, displaying, viaa graphical user interface (GUI), at least the segmented plurality ofcorneal endothelial cells.

Additionally or alternatively, set of operations 100 can comprise one ormore other actions discussed herein in connection with employing atrained DL model to automatically and/or semi-automatically segmentendothelial cells.

Referring to FIG. 2, illustrated is a flow diagram of an examplemethod/set of operations 200 that can be performed by one or moreprocessors to train a model via deep learning to segment endothelialcells, according to various aspects discussed herein.

The set of operations 200 can comprise, at 210, accessing a training setof optical microscopy images of corneal endothelial cells, wherein eachimage can be associated with a ground truth segmentation of itsendothelial cells. In various embodiments and in the example use casesdiscussed below, the images can be obtained via a system and/orapparatus implementing the set of operations 200, or can be obtainedfrom a separate medical imaging system. Additionally, the images can beaccessed contemporaneously with or at any point prior to performing theset of operations 200.

The set of operations 200 can further comprise, at 220, pre-processingeach optical microscopy image of the training set to correct for atleast one of shading or illumination artifacts.

The set of operations 200 can further comprise, at 230, training a modelvia deep learning based on the training set of images and the associatedground truth segmentations of the endothelial cells of each image of thetraining set.

Additionally or alternatively, set of operations 200 can comprise one ormore other actions discussed herein in connection with training a modelvia deep learning to segment endothelial cells.

Referring to FIG. 3, illustrated is a flow diagram of an examplemethod/set of operations 300 that can be performed by one or moreprocessors to determine a prognosis for a keratoplasty based on featurescalculated from segmented corneal endothelial cells, according tovarious aspects discussed herein. Processor(s) can include anycombination of general-purpose processors and dedicated processors(e.g., graphics processors, application processors, etc.). The one ormore processors can be coupled with and/or can include memory or storageand can be configured to execute instructions stored in the memory orstorage to enable various apparatus, applications, or operating systemsto perform the operations. The memory or storage devices may includemain memory, disk storage, or any suitable combination thereof. Thememory or storage devices can comprise—but is not limited to—any type ofvolatile or non-volatile memory such as dynamic random access memory(DRAM), static random-access memory (SRAM), erasable programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), Flash memory, or solid-state storage.

The set of operations 300 can comprise, at 310, accessing an opticalmicroscopy image of corneal endothelial cells of a keratoplasty patient.In various embodiments and in the example use cases discussed below, theimage can be obtained via a system and/or apparatus implementing the setof operations 300, or can be obtained from a separate medical imagingsystem (e.g., specular or confocal microscope, etc.). Additionally, theimage can be accessed contemporaneously with or at any point prior toperforming the set of operations 300.

The set of operations 300 can further comprise, at 320, segmenting atleast a subset of the corneal endothelial cells of the image via a deeplearning model, according to techniques discussed herein. For example,segmenting the corneal endothelial cells can comprise pre-processing theimage, generating an output probability image via the deep learningmodel, generating a binarized output via thresholding, and thinning cellboundaries, each of which can be according to techniques discussedherein.

The set of operations 300 can further comprise, at 330, calculating oneor more features associated with the segmented endothelial cells. Theone or more features can comprise existing measures (e.g., endothelialcell density (ECD), coefficient of variation (CV) of cell area,hexagonality (HEX), etc.), one or more graph features (e.g., local orglobal graph features of any of the types discussed herein, etc.), orone or more other features discussed herein, including novel featuresintroduced herein.

The set of operations 300 can further comprise, at 340, generating aprognosis for the keratoplasty (e.g., of rejection, failure ornon-failure; of one of no adverse outcome, non-rejection failure, orrejection; etc.) based on the one or more calculated features via atrained model (e.g., a machine learning (ML) model such as a LinearDiscriminant Analysis (LDA) classifier, a Quadratic DiscriminantAnalysis (QDA) classifier, a Support Vector Machine (SVM) classifier, ora Random Forest (RF) classifier, etc., another type of model, etc.).

Additionally or alternatively, set of operations 300 can comprise one ormore other actions discussed herein in connection with determining aprognosis for keratoplasty based at least in part on features calculatedfrom segmented endothelial cells.

Additional aspects and embodiments are discussed below in connectionwith the following example use cases.

Example Use Case 1: Assessment of Endothelial Cells and Corneas at Riskfrom Ophthalmological Images

The following discussion provides example embodiments in connection witha first example use case involving analyzing the images of corneaendothelial cells and determination of the degradation of theseimportant cells maintaining corneal clarity.

1. Overview

Review of CWRU's NIH-funded Cornea Donor Study (CDS) with the attachedSpecular Microscopy Ancillary Study (SMAS) and the Cornea PreservationTime Study (CPTS). The first example use case was developed based onenhanced understanding, obtained from the CDS, of many factorssurrounding the success of penetrating keratoplasty (PK) for conditionswith endothelial dysfunction, notably Fuchs dystrophy andpseudophakic/aphakic corneal edema. In the associated SMAS, endothelialcell loss (ECL) was assessed in a subset of participants to examine itsrelationship with donor age. A slight association between increasingdonor age and greater post-PKP corneal endothelial cell loss by 5 yearswas found among eyes whose grafts were successful at 5 years.Endothelial Cell Density (ECD) at 6 months, 1 year and 5 years wasstrongly predictive with subsequent graft failure. The CPTS dataprovides important information related to ECL following Descemetstripping automated endothelial keratoplasty (DSAEK) at three yearsrelated to storage time as well as the impact of donor, recipient,surgical and post-surgical factors on graft success and EC health. Sincethe literature suggests that changes in morphometrics (coefficient ofvariation (CV) of cell area and percentage of hexagonal cells orhexagonality (HEX)) may be more sensitive than ECD in assessingendothelial health and dysfunction, this was examined in connection withthe first example use case in a case-comparison group study from asubset of SMAS participants. It was found that 6-month HEX resultssuggested an association with subsequent graft failure, whereas CV wasnot predictive of graft failure. The first example use case examinedimage analytics, as a more sensitive predictor of graft failure.

Review of image processing solutions. Recent papers demonstrateapproaches in computerized image analysis of EC images as obtained fromspecular microscopes and, less commonly, confocal microscopes. There isopportunity for improved software. High-quality images in publicationslead to concern about performance in clinical practice. Some recentpapers use uncommon confocal rather than specular microscopy. Severalreports describe limited performance of commercial systems clinically,suggesting a need for improvement. Nearly the entire focus has been onECD, CV, and HEX, suggesting an opportunity to uncover new, importantmorphological and cell disarray image attributes, such as thosediscussed herein. Moreover, no quantitative paper has investigated imageattributes associated with keratoplasty rejection.

Opportunity with machine learning. There are many success storiesshowing the ability of medical image analytics to predict clinicaloutcomes in cancer, Alzheimer's, and more. There has been particularsuccess when applying image analytics to histology. In head-to-headcomparisons, some studies show that image analytics can outperformphysician experts, for example, at prediction of brain tumor recurrence,in a study from the Case Western Reserve University (CWRU) Center ofComputational Imaging and Personalized Diagnostics (CCIPD). Given thelinkage between EC images and cornea health, reports on EC quantitativebiomarkers, and promise of machine learning, it is likely that imageanalytics will predict keratoplasty failures much better than currentstrategies.

Data. The quality and quantity of aggregated data allowed for highquality machine learning solutions to be created in connection with thefirst example use case. The data sources included the following. SMAS(EY12728, EY012358) over a 10-year observation window includes 609subjects with 105 failures with 8 imaging time points (6 mo, 1 Y, 2 Y, 3Y, 4 Y, 5 Y, 7/8 Y, and 10 Y), each with 3 image repeats, giving 14,616total images (2,520 failure images), minus drop out or deceasedindividuals. CPTS (EY020798) includes 1330 eyes at 5 time points(baseline, 6 months, 1, 2 and 3 years), each with 3 image repeats.Assuming a 10% failure rate, there are 15,960 total images (1,596failure images). CNIIOS will consist of at least 50 failures and 100matching no-failures, at about 6 time points with 3 repeats. This gives2,700 total images (900 failure images). Altogether, the data comprises31,476 total images with about 5,016 images from eyes with a graftfailure. For the first example use case, focusing on the predictivevalue of 6 month images, there were an estimated 6,267 images with 864images from eyes with eventual graft failure. These images are allwell-curated, manually analyzed images.

Advantages. As demonstrated by the first example use case, variousembodiments can detect keratoplasties at risk of failing and/orrejection, leading to early interventions that would reduce failuresand/or rejections. Saving the initial keratoplasty is quite important,as there is a greater rejection rate in a second keratoplasty.Embodiments can provide multiple advantages, including reduced healthcare costs, patient discomfort, patient angst, and vision loss. Variousembodiments can employ techniques discussed herein using specular(and/or confocal) microscopic images and image analytics to predictcorneas at risk. The first example use case and various embodiments alsointroduce novel concepts and approaches to keratoplasty rejection andfailure prediction. Various embodiments can use powerful image analyticsapproaches to predict corneal grafts at risk. Computational featuresemployed can comprise graph analytics approaches that improve uponcurrent HEX measures by assessing disarray both between neighboringcells and cells in the more distant surrounding area. Other features canobjectively evaluate attributes identified by the NIIOS group in theirvisual scoring system for predicting early rejection.

Motivation for image analytics approach. Normal EC images are comprisedof compact, very regular hexagonal arrays of cells. In nature, thehexagonal structure is preferred to other ones that provide perfectcoverage with thin membranes (only squares and triangles), because thehexagonal structure maximizes the area per cell while minimizing thenumber of cells required to provide coverage, and minimizes length ofcontact to create the “least interface energy”. Hence, there is asubstantive history of computing quantitative biomarkers (e.g., ECD andHEX) to assess cornea health. The first example use case studied graphanalytics that capture the full expression of structural disarray innon-normal EC images. Previously, many have described other EC imageattributes, e.g., intracellular dark circles corresponding to nuclei,intracellular bright punctate corresponding to pigment deposits,intracellular dark structures corresponding to vacuoles, and darkstructures at cell intersections corresponding to invading white bloodcells. Consultants Baydoun and Melles implicated visual attributes withkeratoplasty rejection by visually comparing images from eyes with andwithout rejection. The first example use case computed a number ofmorphometric and intensity-based image features that capture the essenceof these attributes and more. The features analyzed in the first exampleuse case provide very rich inputs from images that would be verydifficult for a person, but easy for a computer, to compare between setsof failure and no-failure images. Results obtained in connection withthe first example use case show that these features can distinguishnormal and abnormal EC images.

2. Techniques

The first example use case comprised several different techniques. Toenable routine, objective evaluation of corneas at risk, processing canbe either fully automated or highly automated. The techniques employedin the first example use case comprise: (1) Segmentation of endothelialcells; (2) Determination of classic EC metrics; (3) Determination ofcellular features; and (4) Determination of corneas at risk. Each ofthese techniques is described in greater detail below.

Referring to FIG. 4, illustrated is an overview of the techniquesemployed in the first example use case, along with an initial andendothelial cell segmented version of an example EC image, in connectionwith various aspects discussed herein. When segmenting EC images, acell-border was defined for the first example use case as a collectionof curve segments enclosing a bright cell. Manual segmentation of animage might take 15-30 minutes. Automatic analysis according to aspectsdiscussed herein can reduce this time to seconds.

FIGS. 5-30, discussed in greater detail below, show example images,graphs, and other information in connection with the first example usecase. FIGS. 5-12 relate to preliminary results during an initial stageof the first example use case. FIG. 5 shows changes in EC images between0.5-3 years as cells are lost after PK. Although the 6-month image showsgood cell density, there is disarray (at 510) as compared to the veryregular hexagonal structure in a normal EC layer (not shown). Amongother features, the first example use case used four types of graphs tocapture cellular structural relationships, as shown at FIG. 6. Delaunaygraphs give clear visual and quantitative differences between sets ofrepeated images in FIG. 7. For these same sets of images, preliminaryanalysis found that 32 of 80 features were statistically different (FIG.8). Various embodiments can employ these features or other featuresdetermined to have feature stability, a highly desirable property,across repeated images. Initial analysis also noted many visualattributes of EC images that are not captured by existing quantitativebiomarkers (ECD, HEX, CV). The first example use case quantitativelyassessed dark nuclei and bright pigments (FIG. 9) as well as thicknessof dark gaps between cells (FIG. 10), using preliminary image processingalgorithms. Visual assessments of such attributes have been related tokeratoplasty rejection by the consultants. FIGS. 11-12 illustratetechniques for determining cell density and hexagonality (HEX). FIGS.13-15 illustrate deep learning techniques for segmenting ECs.

Automated Segmentation of Endothelial Cells

The automated image processing pipeline to generate segmentations ofendothelial cells from the first example use case comprised three mainsteps. First, images were pre-processed to correct forshading/illumination artifacts. Second, a learning algorithm was trainedto generate pixel-wise class probability maps. Finally, thresholding andmorphological processing operations were performed to generate the finalbinary segmentation maps. Each of these steps is described in detailbelow.

EC images, especially specular microscopy images, are commonlyassociated with varying illumination across the imaging area. The firstexample use case corrected this artifact by generating a low-passbackground image using a Gaussian blur and dividing the original imagewith the background image. This resulted in a flattened image with evenillumination across the complete field of view.

In recent years, deep learning methods (especially convolutional neuralnetworks (CNNs)) have been extensively used to perform medical imageanalysis tasks, such as image classification and segmentation. CNNarchitectures are commonly comprised of convolutional, pooling andup-sampling layers and are trained to minimize a pre-defined lossfunction. Convolutional layers learn one or more filters, where theweights of the kernel filter are shared across the image. Pooling layersreduce the size of the feature maps allowing the network to be invariantto small translations. Upsampling layers allow the creation of an outputprobability image with the same size as the input. The U-Net networkarchitecture was used in the first example use case, but techniquesdiscussed herein can be extended to any CNN architecture capable ofperforming either image segmentation or classification tasks.

U-Net was initially shown to segment neuronal structures in electronmicroscopy (EM) stacks with low pixel errors. Since EC images arevisually similar to such stacks (both contain dark cell border regionsbetween brighter cell regions), it was hypothesized that the networkwould perform well for the task of EC segmentation. The network usesskip connections to recover the full spatial resolution in its decodinglayers, allowing one to train such deep fully convolutional networks forsemantic segmentation. In initial experiments for the first example usecase, the network was trained in a ten-fold cross validation fashionover 100 images and tested on a held-out data set of 30 images. Resultsfrom the network on images from the held-out test set are shown in FIG.19 and other figures. Dice and Jaccard coefficients obtained for U-Netusing a fixed threshold and an adaptive threshold are in table 1, below.

TABLE 1 Dice and Jaccard coefficients for CNNs and thresholdingtechniques Cell Basis Metric Minimum Average Maximum Std. Dev. DiceCoefficient 0.00 0.87 0.98 0.17 Jaccard Coefficient 0.00 0.80 0.96 0.18

Finally, the probability maps from the neural network were binarizedusing one of two methods: adaptive Otsu thresholding over a neighborhood⅛^(th) the size of the input image or a simple [7×7] sliding windowlocal thresholding approach. In the resulting binary image, predictedcell borders are labeled white (255) and other regions labeled dark (0).A series of morphological operations are performed on the binary imagesto create thin strokes between cells and to clean the results. Detailsof the morphological operations are described in the section below.

Automated Determination of Existing EC Metrics

After obtaining segmentation results, the post-processing pipelineinvolves binarization, skeletonization, and morphological cleaning.Binarization was conducted via an adaptive Otsu threshold. The adaptivethreshold was calculated using Otsu's threshold selection method withina sliding window approximately ⅛th the size of the segmentationresulting image. The binarized image was inversed so the end product iswhite cell borders with black cells and surrounding area. In otheraspects, binarization can be obtained via an adaptive threshold. Theadaptive threshold was calculated within a sliding window at each pixellocation using the following equation: T=m[1+k(σ/σ_(dyn)−1)], where T isthe adaptive threshold value, m is the mean intensity in the pixelneighborhood, a is the standard deviation in the same neighborhood,σ_(dyn) is the difference between the maximum and minimum standarddeviation across the image, and k is a tunable parameter of thethresholding algorithm. After either technique for thresholding, thebinarized image was inverted so the output comprised white cell borderswith black cells and surrounding area.

Four consecutive morphological operations were employed in the exampleused case to create thin strokes between cells and to clean thebinarized result. First, a morphological closing operation with astructuring element comprising a disk with radius 4 was performed toclose cell borders with gaps from the binarization process. Second, theresult was processed with a thinning operation. Thinning results in1-pixel wide cell borders, thereby matching the width in the groundtruth labels. Third, a flood-fill operation was applied, delineating allcells and borders white, and the surrounding area black. This processleft small erroneous cell border segmentations outside the primarysegmentation regions. A morphological area opening operation wasperformed that identified and removed 4-connected pixels or anyconnected components less than 50 pixels. Finally, this image wasmultiplied by the inverse of the image produced after the secondmanipulation. The result was a binary image with only the cell areapixels white, and all cell border and surrounding pixels colored black.An example of a raw image, its ground truth segmentation, the automatedsegmentation result, the binarized segmentation, and the skeletonizedbinary image are shown in FIG. 13.

In various embodiments, minor segmentation errors can result due toespecially challenging areas within a given image or rare cell and cellborder cases. To improve the automatically generated segmentations,Guided Manual Correction Software was developed in connection with thefirst example use case, and can be used to edit the binarizedsegmentations. Briefly, this software allows one to view multiple imagesrelated to the automatic segmentation. The software highlights cellswithin the binarized segmentation that are deemed to have been segmentedincorrectly by the automatic segmentation algorithm. The user then canerase or edit the cell borders utilizing built-in tools within thesoftware. The final edited binarized segmentation will be used insubsequent processing. The algorithm for identifying potentialsegmentation errors is based upon deep learning probability outputs andcell areas. The Guided Manual Correction Software is discussed ingreater detail in co-owned U.S. Provisional Patent Application No.62/985,481, filed Mar. 5, 2020, and entitled “AUTOMATED SEGMENTATION ANDGUIDED CORRECTION OF ENDOTHELIAL CELL IMAGES,” the contents of which areherein incorporated by reference in their entirety.

To obtain the ECD from the skeletonized binary segmentation result, thetotal cell area was calculated by automatically counting the totalnumber of pixels the cell area (including the cell borders) andmultiplying this by the area of one pixel (0.65 um²). The area of onepixel is obtained from the corresponding microscope's calibrationmeasurements. Then a connected components tool was used to identifyevery cell in the region of interest; two area thresholds were used toremove the background area (>6500 um²) and regions that wereunrealistically small for cells (<65 um²). The remaining cells werecounted and noted. The ECD for each image is the number of cells dividedby the total area of the cell sample region of interest.

CV calculations start with using connected components and areathresholds to identify true cells in the region of interest. Then thearea of each cell was calculated individually including its border.Outliers were removed from the list of cell areas as these are theresult of improper cell closings during the segmentation andpost-processing steps. The standard deviation and mean of the individualcell areas are calculated, and the ratio of these two values representthe CV.

To determine the HEX value of an image, a shape detection algorithm isused to identify hexagonal versus non-hexagonal cells. The skeletonizedimage mentioned previously is inverted, then the white borders weredilated by 2 pixels, and finally reversed again so that the final imagehas white cell area with black thick borders and black surrounding area.The shape detection algorithm utilized in the first example use case wasbased on the Ramer-Douglas-Peucker algorithm, which takes a set ofpoints in a curve, and reduces the points to define a similar curve orline with less points, and is applied to the cell borders. Using acontour approximation, vertices were defined at the end of thesecurves/lines. The number of vertices per shape, or cell in this case,are counted, and if there are 6 vertices, the cell is labeled a hexagon;if there are not 6 vertices, the cell is labeled non-hexagon. HEX equalsthe number of hexagonal cells divided by the total number of cells inregion of interest. FIG. 14 displays an example result of this shapedetection algorithm.

Determination of Other Cellular Features

Hundreds of other features were computed from the segmented EC images.First, the structure of cellular arrangement was captured using graphs.Graph analytics were applied (e.g., Delaunay and cell cluster graphs,and their computed features, as shown in FIGS. 6-8). Graph analyticfeatures comprises features for each node and/or global features. Asthere are a very large number of potential features, various featureselection techniques (e.g., minimum-redundancy-maximum-relevance (mRMR),etc.) were applied to identify the ones most important for determiningcorneas at risk. Second, cellular architectural morphometric measureswere computed from single cells (e.g., area and eccentricity, etc.), inmany cases using existing library computer functions. Third, a varietyof hand-crafted features targeting image attributes in the literaturewere computed (e.g., architecture features shown in FIG. 8). Thesefeatures were developed based upon image attributes related to rejection(e.g., irregular cell morphology, irregular distribution, cell nuclei,cellular reflectivity, wall thickness, etc.). Additionally, examples ofnew measures (wall thickness, bright pigment, and dark nucleus) areillustrated in FIGS. 9-10. For most features, statistical measures overthe image (e.g., mean, standard deviation, skewness, kurtosis, and/orbinned histogram, etc.) were extracted.

Determination of Corneas at Risk

Machine learning classifiers were trained on large sets of data whichinclude EC images as well as outcomes (e.g., rejection, failure or noadverse event) and used for validation on held-out test sets. Softwaredeveloped for the first example use case segments ECs, extracts featuresas discussed above, and predicts from the EC images those cornea(s) thatwill experience a rejection episode or future graft failure. In additionto predicting keratoplasty outcome, various embodiments can evaluaterate of cell loss and time to event as a secondary outcome. Machinelearning classifiers and classification results of various embodimentscan provide highly accurate results, based on training and testing onmore than 6,000 well-curated EC images obtained at multiple time pointsfrom already-collected corneal datasets, including SMAS (EY12728)applied to penetrating keratoplasty (PK) at CWRU, CPTS (EY020798)applied to Descemet stripping automated endothelial keratoplasty (DSAEK)at CWRU, and corneal EC images from the Netherlands Institute forInnovative Ocular Surgery, hereafter called CNIIOS, applied to Descemetmembrane endothelial keratoplasty (DMEK).

Details. The actual classifiers can be created using support vectormachine (SVM), random forest, or any number of other classificationmethods (e.g., a Linear Discriminant Analysis (LDA) classifier, aQuadratic Discriminant Analysis (QDA) classifier, etc.). Classifierswere built on images from multiple time-points simultaneously, 6-monthpost-keratoplasty images, and serial time-point images. To reduce thetotal number of features and improve generalizability, techniques forfeature reduction were employed. Feature reduction methods (for example,minimum redundancy maximum relevance (mRMR)) were applied to reduce thenumber of features considered and to possibly suggest features forpotential additional refinement. Another dimensionality reductionapproach would be to use a Cox Proportional Hazards Model (CPHM) or theWilcoxon rank-sum test to determine features most indicative of a futureadverse event (rejection or failure). Performance was assessed usingcross-validation experiments and the number of trials was increased byrotating fold membership. Classifiers can be tuned appropriately. Sincethe number of no-adverse events greatly exceeds the number of rejectionepisodes and graft failures in the datasets, steps can be taken fortraining with imbalanced data. Feature stability was assessed across the3 image repeats using intra-class correlation coefficient and optionallya latent instability score championed by others. If deemed appropriate,a second feature stability test was run comparing good and fair qualityimages routinely labeled by CIARC. In other embodiments, deep learningtechniques can be employed in place of machine learning classifiers todistinguish EC images between no-adverse event vs. failure and/orno-adverse event vs. non-rejection failure vs. rejection.

Additional work in connection with the first example use case comprisescollecting over 900 images from the NIIOS dataset. Based on theseimages, automatic and guided cell segmentation can be conducted. Basedon the segmented images, features can be calculated, which can comprisestandard features, graphical features, and/or novel features. From thecalculated features, classification experiments can be performed on theEC features to determine the ability to classify keratoplasties prone torejection.

FIGS. 5-27, discussed below, show example images, graphs, and otherinformation in connection with the first example use case.

Referring to FIG. 5, illustrated are serial specular microscope imagingfollowing a PK procedure, in connection with various aspects discussedherein. Images 510-540 were taken at 6 months, 1 year, 2 years, and 3years, respectively, and show a continuing decrease (2684, 2027, 1430,1184 cells/mm², respectively) in cell density. This eye later becamedysfunctional. Notably, although there is a high density of cells at 6months, there is variation in cell size and irregular non-hexagonalarrangement, showing disarray as compared to the very regular hexagonalstructure in a normal EC layer. Various embodiments can predict eyes atrisk from such images using machine learning.

Referring to FIG. 6, illustrated are example images showing four typesof graph structures that can be employed to capture cell arrangement, inconnection with various embodiments discussed herein. Cell centers weremarked in the EC image 610. In 620-650, the graphs shown are Delauney,Voronoi, minimum spanning tree (MST), and cell cluster graph (CCG),respectively. Many computational features were computed from the graphs,for example, average area from Voronoi, average eccentricity from cellcluster graph, standard deviation of edge length from MST, etc.

Referring to FIG. 7, illustrated are example images showing graphanalysis of two groups (top and bottom rows) of centrally located,repeated EC images, in connection with various aspects discussed herein.Delauney graphs show clear visual differences between the two groups.Features from the graphs showing statistically significant differences(p<0.05, as shown in FIG. 8) included standard deviation of trianglearea from Delauney, average chord length from Voronoi, disorder of edgelength from MST, and percentage of isolated points from CCG.

Referring to FIG. 8, illustrated is a graph showing statisticallydifference features (those above the 0.95 line) as computed from twosets of repeated, central EC images as in FIG. 7, in connection withvarious aspects discussed herein. Features shown in FIG. 8 were computedfrom 4 graph types (as in FIG. 6) and from tissue architecture (e.g.,average distance to 7 nearest neighbors, although a different number ofneighbors can be used in various embodiments). A permutation test wasused to evaluate significance. Of the 80 features in the categories ofFIG. 8 that were evaluated, 32 were significantly different (p<0.05) andmore were nearly significant. FIG. 8 shows examples of features that canbe created that show differences between groups of EC images. Onecriterion for good features is stability across repeated images, andadditional, less preliminary results are discussed below. Variousembodiments can employ any of a variety of features discussed herein forclassification of EC images (e.g., as normal vs. failure risk, or normalvs. non-rejection failure risk vs. rejection risk, etc.).

The first example use case identified many visual attributes of ECimages that are not captured by existing quantitative biomarkers (ECD,HEX, CV). Referring to FIG. 9, illustrated are images showing imageprocessing to identify dark nuclei (910-930) and bright pigments(940-960) in EC images, in connection with various aspects discussedherein. For dark nuclei, starting with segmented cells, intensitybackground variation was corrected and noise reduced (920, 950), andconditional dilation was applied to a threshold. Small regions wereremoved. Bright pigments (940-960) were processed similarly. Referringto FIG. 10, illustrated are a graph and heatmap of an example EC imageshowing variations in thickness of dark gaps between cells, inconnection with various aspects discussed herein. On the right of FIG.10 is shown a heatmap of “dark gaps” between cells. Starting withsegmented cells, the full width of the half-intensity-minimum of thecurve was measured, as shown on the left of FIG. 10. The curve is anaverage of 11 perpendicular lines extracted from the midway point of aside. Gaps can be accentuated by lateral distances as well aspotentially cell height in these obliquely illuminated images. Visualassessments of attributes such as those shown in FIGS. 9-10 have beenrelated to keratoplasty rejection by the consultants.

FIGS. 11-12 show techniques related to determining endothelial celldensity (ECD) and hexagonality (HEX). Referring to FIG. 11, illustratedare example images associated with a post-processing pipeline of thefirst example use case, in connection with various aspects discussedherein. FIG. 11 shows the following in connection with a post-processingpipeline for an example image: (a) Corneal endothelium image capturedvia specular microscopy at 1110; (b) ground truth segmentation at 1120;(c) U-Net segmentation probability result at 1130; (d) binarizedsegmentation at 1140; and (e) skeletonized segmentation at 1150.Referring to FIG. 12, illustrated are example images showinghexagonality determined based on techniques of the first example usecase, in connection with various aspects discussed herein. FIG. 12 showsthe following results: (a) Corneal endothelium image and corner analysiswith borders in green at 1210; (b) probabilities from the deep neuralnetwork at 1220; (c) results of algorithm on binarized network outputswith H=hex and NH=non-hex at 1230.

Referring to FIG. 13, illustrated is a diagram showing an overview oftraining, testing, and application of the deep learning model forendothelial cell segmentation, in connection with various aspectsdiscussed herein. The deep learning segmentation algorithm in the firstexample use case started with 100 EC images and corresponding manualannotation labels to train a deep learning network called U-Net. U-Netcan learn features from the training images and labels so it canclassify a pixel as either cell border, or cell and surrounding area.Once trained, additional EC images (e.g., from a test set, patientimaging, etc.) can be provided to the EC segmentation learned network togenerate segmented results. For the first example use case, of 130 ECimages, 30 were in the held out test set, and the remaining 100 wereused with 5-fold cross validation to train the model for subsequenttesting on the 30 held out test images.

Referring to FIG. 14, illustrated are example specular microscopy images1410-1460 with varying endothelial cell density, durationpost-transplant, and contrast from left to right across the image area,in connection with various aspects discussed herein.

Referring to FIG. 15, illustrated is a diagram showing an example U-Netarchitecture that can be employed in connection with various aspectsdiscussed herein. In FIG. 15, the numbers on top and the left bottomcorner of each tensor indicate the number of channels/filters and itsheight and width respectively. The network comprises multiple encodingand decoding steps with three skip connections between the layers (whitearrows). The remaining arrows are as follows: the blue rightward arrowis a convolution with (3, 3) filter size, ReLU activation; the upwardarrow is an up-convolution with a (2, 2) kernel using max poolingindices from the encoding layers; the downward arrow is a max poolingwith a (2, 2) kernel; and the pink rightward arrow with a “1” in thearrowhead is a (1, 1) convolution with sigmoid activation.

Referring to FIG. 16, illustrated are images showing two differentthresholding techniques that can be employed to convert probabilityoutputs to binarized outputs, in connection with various embodiments.Sliding window adaptive thresholding (1620, comprising 1621-1629) andadaptive Otsu thresholding (1630, comprising 1631-1639) were used toconvert probability outputs from the U-Net algorithm (e.g., exampleprobability output image 1610) to binarized, single-pixel width cellborders (e.g., examples 1629 and 1639) which can be compared to groundtruth labels (e.g., example ground truth labeling 1640). For adaptivethreshold, the steps are: binarization (1621), thinning to obtain singlepixel wide lines (1623), filling and opening to remove extraneousbranches (1625), multiplication with 1623 to obtain the cells bordersthat fully enclose cells (1627), and finally pruning to remove smallspurious segments (1629). Image 1629 was compared to the ground truthlabel image 1640 for training and testing. Adaptive Otsu (1630) wasprocessed the same way, at 1631-1639.

Referring to FIG. 17, illustrated are images showing an exampleendothelial cell image, manual cell segmentation analysis, and thecorresponding ground truth, corrected for border overlaps, used in thefirst example use case, in connection with various aspects discussedherein. At 1710 is a good quality sample EC image with cells of variousshapes and sizes, showing the consistently-obtained brightness variationfrom left to right. At 1720 is the cell-border segmentation overlaycreated by an analyst using the HAI CAS software. In this good qualityimage, segmentations were obtained throughout the image (cells cut-offat the boundaries of the image are not labeled). At 1730 is the binarysegmentation map used as a ground-truth label image. The arrows indicateexample double borders generated by the HAI-CAS software in 1720 thatwere replaced with single borders in 1730 following processing).

Referring to FIG. 18, illustrated are example images showingpre-processing shading correction methods applied to a representative ECimage, in connection with various aspects discussed herein. The originalimage 1810 is corrected via localization (1820), division (1830), andsubtraction (1840) methods. Each of the correction methods significantlyflattened the image. For the division method, a low-pass backgroundimage was generated using a Gaussian blur and divided the original imageby the background image to create a flattened image. A normalizedGaussian filter with standard deviation σ=21 pixels and a kernel footprint (65×65 pixels) was used, the 99th percentile of the flatteneduint8 image's intensity was set to the maximum intensity (255), and thenthe rest of the image's intensity was normalized so its distributionexpanded the (0 to 255) intensity range. For the subtraction method, alow-pass background image was generated via the same Gaussian blurpreviously mentioned, and then this background image was subtracted fromthe original image, wherein the flattened image was normalized using thesame method as before. For the localization method, the brightness wasnormalized along the vertical and horizontal directions by adding thedifference between the average image brightness and the averagebrightness in the corresponding row/column at each pixel. New pixelvalues were given as

${{p^{\prime}\left( {x,y} \right)} = {{p\left( {x,y} \right)} + \left\lbrack {L - {\sum_{j}\frac{p\left( {x,j} \right)}{H}}} \right\rbrack + \left\lbrack {L - {\sum_{i}\frac{p\left( {i,y} \right)}{W}}} \right\rbrack}},$

where p′(x,y) is the new pixel value, p(x,y) is the original pixelvalue, L is the average brightness in the image, H is the height of theimage, and W is the width of the image.

Referring to FIG. 19A, illustrated are images of deep learningprobability maps of an example test image following four types ofpre-processing, in connection with various aspects discussed herein. Anoriginal test image 1900 was processed with deep learning following nopre-processing 1902, localization pre-processing 1904, divisionpre-processing 1906, and subtraction pre-processing 1908. In the centerof the probability outputs, the borders appear more black due to thehigh confidence of the trained algorithm in these regions, however,toward the edges and some blurry areas of the original images, theborders appear more gray because the algorithm is less confident here(indicated by arrows).

Referring to FIG. 19B, illustrated are images showing the finalsegmentation results of the EC image of FIG. 18 with the fourpre-processing methods discussed above, according to various aspectsdiscussed herein. At 1950 is the ground truth label. The finalsegmentation results shown in 1952-1958 are: after no pre-processing(1952), after localization pre-processing (1954), after divisionpre-processing (1956), and after subtraction pre-processing (1958). Thedashed circles indicate correct cell segmentation, whereas the solidcircles indicate incorrect cell segmentation. The arrows indicateregions of newly segmented cells not previously annotated.

Referring to FIG. 20A, illustrated are three example EC images (2000,2004, and 2008) and corresponding segmentation (2002, 2006, and 2010)highlighting cells with differences between manual and automaticsegmentations, in connection with various aspects discussed herein. Cellboundaries solely identified via manual segmentation are in green, thoseidentified via both are in yellow, and boundaries solely identified viaautomatic segmentation are in red. Images 2000 and 2002 shows twoexamples in white circles where the automated method first split asingle cell and then merged two cells into one cell. The reviewersagreed the manual annotations were more accurate than the automatedsegmentations. Images 2004 and 2006 illustrates an example where theautomated method merged two cells into one cell, and the reviewersconcluded the case equivocal because it was unclear which methodactually segmented the cell correctly. Images 2008 and 2010 shows anexample where the automated method split one of the manually annotatedcells into two cells. The reviewers considered this case to have beensegmented more accurately by the automated method than the manualannotation.

Referring to FIG. 20B, illustrated are three examples of EC imagesshowing the automatic segmentation method identifying and segmentingcells outside of the sample area of the ground truth annotations, inconnection with various aspects discussed herein. Images 2050, 2054, and2058 are sample EC image from the independent held-out test set, andimages 2052, 2056, and 2060 show the respective ground truth (green) andautomatic (red) segmentations overlaid, showing cells outside of theground truth sample area segmented by the U-Net algorithm.

Referring to FIG. 21, illustrated are charts showing the results ofvisual analysis of automated cells in a held-out test set comprising 30images, in connection with various aspects discussed herein. Within themanually analyzed regions, experts identified that only 52 of 1876 cells(3%) required reconsideration. A histogram of numbers of discrepanciesare shown at 2110, where 22 images (73%) contained 0, 1, or 2 potentialdiscrepancies. Notably, among discrepancies, the automated result wasdeemed better or equivocal 16% of the time, giving only 44 (2.3%) ofcells where experts deemed it desirable to change the automated result,as seen at 2120. In the regions newly segmented by the automated method,564 new cells (an increase of 30%) were identified, with the imagehistogram shown at 2130. Nearly 70% or 390 of newly segmented cells weredeemed acceptable (accurate or equivocal in the pie chart at 2140).

Referring to FIG. 22, illustrated are images showing three examplepost-DSAEK EC images (corresponding to the three rows) having undergoneboth manual and automatic segmentation, in connection with variousaspects discussed herein. Column 2210 shows the original, raw EC images.Column 2220 shows the enhanced, pre-processed EC images. Column 2230shows the ground truth images corrected. Column 2240 shows the U-Netprobability outputs. Column 2250 shows the final border segmentationsfollowing post-processing pipeline. Column 2260 shows an overlay ofground truth manual segmentations (green) and automatic segmentations(red). Yellow indicates where the manual and automatic segmentationsoverlapped.

Referring to FIG. 23, illustrated is an example screenshot ofsemi-automated segmentation graphic user interface (GUI) software, inconnection with various aspects discussed herein. In the large viewingwindow, the GUI displays the enhanced EC image with the automaticsegmentations overlay. On the side, the user can see the original raw ECimage, the flattened, enhanced EC image, and the probability output tocross reference information and to aid during the manual editingprocess. Below the large viewing window, the user has the option toselect a pen for adding cell borders, and two sizes for erasing cellborders. There is also a button to toggle between the automatic bordersegmentation and the highlighting of cells that may require a secondreview.

Referring to FIG. 24, illustrated are example images showing cells withoutlier areas or cells that were under-segmented and/or had faint orincomplete borders noticeable in the corresponding cell's area of theprobability output, in connection with various aspects discussed herein.Circles indicate cells that matched these two conditions. At 2410 is theU-Net probability output. At 2420 is the final automatic segmentations.At 2430 is the enhanced EC imaged with automatic segmentation overlayand erroneous cells highlighted for second review. At 2440 are manualedits applied to automatic segmentation using the pen tool in the GUIsoftware.

Referring to FIG. 25, illustrated are Bland Altman plots for ECD, CV,and HEX, in connection with various aspects discussed herein.

Referring to FIG. 26, illustrated is a heatmap of an example EC image,showing border thickness, in connection with various aspects discussedherein. Border thickness was measured from EC images because it isbelieved that when the cornea swells, the cell borders of theendothelium will thicken.

Referring to FIG. 27, illustrated is a confusion matrix showingpredicted and actual classification results for a machine learningclassifier of the first example use case, in connection with variousaspects discussed herein. A small data cohort of 28 patients with 633 ECimages was used to train a random forest machine learning classifier(with 5-fold cross validation, as discussed above) to predictkeratoplasty rejection. From the confusion matrix, out of 371 EC imagesfrom rejection eyes, the model accurately classified 329 images asrejection, giving a sensitivity of 0.8868, and correctly identified 192of the 262 control images, for a specificity of 0.7328.

Once trained, the algorithm can be deployed to analyze EC images for therisk of failure or rejection. All training can be encoded in a deployedmachine learning model. Various embodiments can be deployed on astand-alone computer or distributed over the internet from a server.Additionally, as more data becomes available, learning algorithms can betrained on more data, and new model(s) created for deployment.

Example Use Case 2: Semi-Automated Segmentation of Corneal EndothelialImages

The following discussion provides example embodiments in connection witha second example use case involving automated and semi-automatedsegmentation of corneal endothelial images via a trained deep learning(DL) model.

Abstract

Corneal endothelial cell (EC) layer images are utilized to indirectlyassess the health of the cornea post-transplantation. Cellular,morphological biomarkers of cornea health are computed from theseimages, but first they require manual or automatic segmentation of cellsand cell borders. As manual segmentation is very time consuming, thesecond example use case developed a semi-automated approach. A deepneural network architecture (U-Net) was used to segment cells inspecular microscopy images of the corneal endothelium, with a 0.87 Dicecoefficient on a per image basis. A visual study analyzed newlyidentified and split/merged cells in the automatic segmentation resultsas compared to annotations. It was determined that 53 (3%) of the 1,867cells in 30 test EC images benefited from an additional manual edit. Ofthe 30 test images, 6 had no edits, 20 had 1-3 edits, and 4 had 4-5edits. Based on these results, any new image is likely to have 1-3discrepancies between the manual and automatic segmentation. Thesesegmentation differences generally occur in the dark areas or regionswith unclear borders in the images, and are easily fixed with theaddition or deletion of a border. Interactive software was designed inconnection with the second example use case that highlighted specificborders/areas, where segmentations are suspicious, for potentialediting. There are multiple criteria for suspicious cells (e.g., outliercell areas with respect to the average cell area in a given image andunder-segmented cells with visible faint or disconnected borders in theDL (e.g., U-Net) output images). This approach can allow clinicians andexpert analysts to analyze images in 1-2 minutes as compared to 20minutes now, likely with reduced inter-reader variability. This willenable efficient and precise research studies involving several hundredsof images in the prediction of corneas at risk.

1. Introduction

For the second example use case, an automatic segmentation method wasdeveloped to delineate the dark cell borders of endothelial cell (EC)images following a corneal transplant, one of the most commontransplants performed with over 44,000 surgeries annually. A healthy EClayer maintains a clear cornea via an active ion-pump mechanism, whichredistributes fluid from the anterior chamber through the cornea. Thesehealthy cells are usually hexagonally shaped, with similar area, and areuniformly aligned without large gaps in-between cells. Over timehowever, corneal ECs start to die but are never replaced by dividingcells. Instead, surrounding ECs morph, compromising their uniform shapeand size. This hinders the ion-pumps mechanism, causing fluid to buildup in the cornea turning it opaque-like and deteriorating vision.Treatment options include topical steroids, but a more successful optionis undergoing a keratoplasty. There are three types of keratoplastytreatment available: penetrating keratoplasty (PK), Descemet strippingautomated endothelial keratoplasty (DSAEK), and Descemet membraneendothelial keratoplasty (DMEK). Each transplant varies based on howmuch of the donor's cornea is extracted and implanted into therecipient's eye.

Following a transplant, clinicians perform regular follow-ups, analyzingthe cornea and the EC layer ensuring clarity and overall health.Clinicians commonly use quantitative biomarkers, endothelial celldensity (ECD), coefficient of variation (CV), and percent hexagonality(HEX), of the EC layer to evaluate cornea health post-transplantation.Briefly, ECD is the number of cells per total sample area of cells inthe image, CV is the standard deviation of cell area divided by the meancell area within the image, and HEX is the percentage of cells that havesix sides. However, in order to calculate these morphometrics, the cellsand their borders need to be identified in the EC layer. This is madedifficult due to the various cell sizes and shapes, varyingillumination, unclear cell and border distinction, and sometimes poorimage quality. The delineation of cell borders can be done manually, orautomatically. Manual segmentation is a laborious process, and there isincreased risk of inter-reader variability when it comes to makingdecisions about classifying an uncertain region. Automatic segmentation,such as watershed algorithms, genetic algorithms, and region-contour andmosaic recognition algorithms, while much quicker and more consistent,may still make errors when trying to segment the uncertain regions. Deeplearning (e.g., U-Net neural network, etc.) has shown promising resultsfor segmenting corneal ECs. These studies have been trained andconducted on various datasets such as non-diseased eyes, glaucomatouseyes, or with cell centroid-marked ground truths.

For the second example use case, a semi-automatic deep learning approachwas developed to segment the cell borders of EC images post-DSAEK. Thisapproach starts with applying a U-Net learning system convolutionalneural network to train and segment 130 specular-microscope, clinicalquality, post-transplant EC images acquired in the Cornea PreservationTime Study (CPTS). However, as mentioned previously, automaticsegmentation may make a couple of errors, thus an automatic segmentationguided correction graphical user interface (GUI) software was alsocreated for users to directly address segmentation errors by eitheradding or erasing cell borders. The GUI software also aids users byhighlighting cells with borders that are potentially incorrect. Thesemanual edits would likely involve 1 to 2 additional minutes after theautomatic segmentation step.

2. Image Processing and Analysis

The deep learning architecture U-Net is employed to classify each pixelin the EC images into one of two classes: cell border or other.Probability outputs from the network show areas of high and lowclassification confidence. A post-processing pipeline was applied to thesegmentation probability outputs following the classification step tobinarize, thin, and prune the cell borders until each border encompassesa full cell. Alongside conducting a quantitative analysis to evaluatethe deep learning segmentation performance, a visual analysis wascarried out. An ophthalmologist and expert reader analyst evaluated eachtest image cell-by-cell difference between the ground truth andsegmentation results to determine which method (automated or manual)accurately segments the cell borders. The results of the visual analysisindicate that a couple manual edits can improve the cell bordersegmentation on most images.

2.1 U-Net Deep Learning Segmentation

The U-Net deep learning segmentation begins with pre-processing the ECimages to remove the illumination gradient that exists across the imagesfrom left to right. Referring to FIG. 28, illustrated are five examples(2810-2850) of clinical quality EC images taken at various time pointsfollowing DSAEKs with this illumination gradient, in connection withvarious aspects discussed herein. To remove this gradual change fromdark to light intensity, a Gaussian filter was applied with standarddeviation s=21 and kernel footprint (65×65 pixels), producing abackground image. The original EC image was divided by this backgroundimage, and then the 99th percentile of the image's intensity was set to255 before normalizing the rest of the image's intensity within therange (0, 255).

After pre-processing the EC images, the U-Net neural networkarchitecture shown in FIG. 15 was used, which has shown promisingresults with regards to image segmentation. U-Net has many advantageousfeatures such as its decoding (downsampling), encoding (upsampling), andskip connections, allowing the network to recover full spatialresolution in its decoding layers. The architecture contained 16 layersand has a receptive field of size (93, 93). U-Net produces probabilityoutput images with pixel values between 0 and 1. Values closer to 0indicate strong confidence of a cell's border, and values closer to 1indicate strong confidence of a cell or other surrounding.

Following U-Net, the probability output images underwent apost-processing pipeline to binarize and clean up the cell borders untilthe result was a binary image with single pixel-width black borders thatfully enclosed white cells. This pipeline started with first upscalingthe probability outputs from values (0 to 1) to create (0 to 255)grayscale images. Then the images were inverted before adaptive Otsuthresholding to binarize the probabilities. The window of the adaptivethreshold was approximately ⅛th the size of the image. This binary imagewas then inverted so the cell borders were white, and thecells/surrounding pixels were black. The image then underwent fivemorphological operations to create thin borders between cells and toclean the binarized result. First, a morphological closing operation wasperformed with a structuring element consisting of a disk with radius 4to close cell borders with gaps from the binarization process. Second,the result was processed with a thinning operation. Thinning results in1-pixel wide cell borders, thereby matching the width in the groundtruth labels. Third, a flood-fill operation was applied, delineating allcells and borders white, and the surrounding area black. This processleft small erroneous cell border segmentations outside the primarysegmentation regions. Fourth, a morphological area opening operation wasperformed to identify and remove 4-connected pixels or any connectedcomponents less than 50 pixels. This image was multiplied by the inverseof the image produced after the second manipulation. Finally, the fifthmorphological operation, a pruning operation, was applied to the productimage to remove spurs and stray lines prior to inverting the image oncemore. The result was a binary image with single-pixel width blackborders that were closed and fully encapsulated white cells, with whitesurrounding area.

2.2 Visual Analysis

A quantitative approach was applied to the 30 held-out test images ofthe dataset, calculating the Dice coefficient and Jaccard index.However, this may not tell the full story of the segmentation process,because sometimes the automated method would segment cells outside theregion of the manually segmented region. There was no way to evaluatethe accuracy of the newly identified cells outside the ground truthregion. Furthermore, there were a few over or under-segmented cellswithin the ground truth region where it was difficult to determine whichmethod accurately segmented the cell border. This was more common whencell borders were obstructed by bright and dark spots in the imagingtechnology. Thus, a visual analysis study was performed where anophthalmologist and expert EC reader analyst evaluated each image on acell-by-cell basis answering the following two questions: (1) Whenautomated and manual cell detections differ (e.g., automated methodidentifies two cells and manual identifies one cell or vice versa),which method (manual, automatic, or an equivocal case) correctlysegmented the cell (The number of discrepancies per image when answeringthis question was recorded as well) and (2) For each newly identifiedcell, was it automatically segmented accurately, inaccurately, or is itan equivocal case.

2.3 Semi-Automated Segmentation GUI

Software was developed to enable rapid manual editing following theautomatic U-Net deep learning segmentation discussed above. Since U-Netmay make a few mistakes when segmenting cell borders in challengingimages, guided correction software was created whereby an operator canquickly identify suspicious cells for correction. A graphical userinterfaced (GUI) was created in the Python programming language version3.7. The software displays four images so the user can cross referencedifferent information from each image in order to make an informeddecision regarding where to add or erase a new or erroneous cell border,respectively. The four corresponding images include the original raw ECimage, the enhanced, pre-processed EC image, the probability maps fromthe U-Net segmentation, and the final segmentation border image afterpost-processing overlaid on the enhanced, pre-processed EC image. TheGUI has a pen and eraser tool for editing the segmentation image.

Two conditions were implemented to identify suspicious cells for editingin each image. Suspicious cells were defined as over-segmented (when onetrue cell has been split into two or more cells) or under-segmented(when two or more cells have been merged into one cell). The automaticsegmentation produced more under-segmented cells than over-segmented.

The algorithms for identifying under-segmented cells follow. First,under-segmented cells can be identified by excessively large areas incomparison to the rest of the cells in the image. Thus, a first way toidentify suspicious cells calculated the area of each cell in an imageand identified outlier cell areas based on those cells having an areaexceeding an associated threshold (e.g., identifying outliers as havingarea greater than 3 standard deviations higher than the mean cell areain the image). Second, under-segmented cells can be identified byaltering the local threshold value within a cell prior to binarizing theU-Net probability output. If the change in local threshold value leadsto two or more cells instead of a single cell, the segmented cell ishighlighted for a second review. The intuition behind this secondcondition is that there are borders in the U-Net probability outputsthat may not be as dark as nearby cell borders, or they aredisconnected. When the adaptive threshold is determined for a pixel'sneighborhood, the fainter borders' intensities fall above the thresholdvalue. Thus, during binarization these faint borders are classified ascells instead of borders and the resulting cell is under-segmented.

To identify over-segmented cells, cells were highlighted that have muchsmaller areas than the average cell area. Similar to theunder-segmentation approach, suspicious cells were identified that haveexcessively small areas in comparison to the rest of the cells in theimage. The area of each cell in the image was calculated, and cells withareas that are lower than an associated threshold (e.g., cells with areagreater than 3 standard deviation lower than the mean cell area, etc.)in the image were identified as potentially over-segmented.

3. Experimental Methods 3.1 Labeled Dataset

The EC images used in this study were retrospectively obtained from theCornea Image Analysis Reading Center (CIARC) along with theircorresponding corner analysis performed in HAI CAS/EB Cell AnalysisSystem software (HAI Laboratories, Lexington, Mass.). A subset of 130images were randomly selected from the CPTS clinical research study,which acquired EC images from 1330 eyes taken at various time pointsbetween 6- to 48-months post-DSAEK. All images used were size (446×304)pixel with pixel area of 0.65 μm² and were taken with Konan specularmicroscopes (Konan Medical, Irvine, Calif.). Each image containedbetween 8-130 cells that were manually identified and their borders weresegmented. The EC densities of these images ranged from 600 to 2450cells/mm². All images were deidentified and handled in a method approvedby the University Hospitals Cleveland Medical Center InstitutionalReview Board.

All images were manually analyzed using the standard operatingprocedures of the Reading Center. Trained readers from the ReadingCenter utilized the HAI corners method to manually segment the cellborders of the raw EC images. The readers mark cell corners with the HAICAS/EB software (HAI Laboratories, Lexington, Mass.) and then thesoftware generated cell borders connecting the corners. FIG. 17,discussed above, shows the manual segmentation applied to an example ECimage. Note in the second image that occasionally the green cell bordersegmentations overlap. To remedy this, the segmentations were dilatedand thinned to created single pixel width borders.

3.2 Classifier Training and Testing

For training the U-Net architecture, 100 images from the 130 EC imagedataset was used for training and the remaining 30 images for testing. A10-fold cross validation to training approach was used, where 90 imagesof the training dataset were used for training, and 10 images were usedfor validation. Training stopped when validation loss performance on thevalidation set did not improve.

Prior to the training process, the EC images were padded on all sides ina symmetric fashion to ensure the convolutions are valid at the edges.U-Net was trained for a maximum of 200 epochs, using weighted binarycross entropy as its loss function. Class imbalance could be accountedfor by weighting the loss function by the inverse of the observed classproportions. For example, in the second example use case, cell bordersoccurred at a low frequency across all EC images (about 5% of thepixels) whereas cells or other accounted from about 95% of the pixels.Therefore, cell border pixels will have a larger weight in thecomputation of the lass function. The network was optimized using theAdam optimizer with an initial learning rate of 1.0e-4. Finally, dataaugmentation was utilized to ensure good generalization performance bythe network.

Briefly, the augmentations used were in the range of 5% translations inheight and width across the image, 5% zoom, 3 degree shear, and randomhorizontal flips. The model with the lowest validation-loss value duringthe training phase was applied to the 30 image held-out test set.

Software for image pre-processing and binarizing the network predictionswere implemented using MATLAB R2018a. U-Net was implemented using theKeras API (with Tensorflow as backend), the Python programming language.Neural network training was performed using two NVIDIA Tesla P100graphics processing unit cards with 11 GB RAM on each card.

4. Results

FIG. 22, discussed above, shows results of the automatic segmentationalgorithm on three held-out test EC images. In the overlay images, thereare red cells outside the ground truth cell region in green, which isindicative of the algorithm's ability to segment newly identified cells.Additionally, there is the occasional cell or two where there is adiscrepancy between the manual and automatic segmentation in whitecircles. These white circles correspond to the red circles seen on theprobability output images where a faint or disconnected border isvisible within the cell's interior region. These cells and areas areexample cases of challenging or uncertain regions that the software canhighlight for a manual follow-up correction.

The quantitative results from the U-Net automatic segmentation (prior tomanual editing) and visual analysis study were as follows. The manualsegmentations found 1876 cells in the 30 test images. Whenquantitatively comparing these cells to the corresponding cells of theautomated segmentation, the average cell Dice coefficient across all 30test images was 0.87. The automated segmentations found an additional507 cells. The expert analyst and ophthalmologist declared 293 (57%) ofthe new cells accurate, 93 (18%) cells inaccurate, and 123 (24%) cellsequivocal. There were 53 (3%) cell discrepancies between the manual andautomated segmentations of the 1876 manually graded cells. The manualsegmentation correctly segmented 44 (83%) of the discrepancies, whilethe automatic segmentation method correctly segmented 1 (2%) of themleaving 8 (15%) of the discrepancies as incorrectly segmented by bothmethods.

Manual editing can be accomplished via the semi-automated segmentationGUI developed and shown in FIG. 23, discussed above. The GUI loads fourimages for a single eye: the original raw EC image, the pre-processedimage, the probability output image, and the overlay of the automaticsegmentation results on the pre-processed image. It was determined thatthe main viewing window should display the enhanced EC image instead ofthe raw image because the enhanced image can show cells more clearly inthe darker regions of the original image. In the top left corner of theenhanced and the original images in FIG. 23, the cells are clearer andtheir borders are more apparent in the enhanced EC image.

As mentioned previously, there were less than 3% of cell discrepancies,or suspicious cells as described earlier, between the manual andautomatic segmentations. Some of these discrepancies were due to faintborders in the probability output images that had higher intensitylevels compared to neighboring cell borders, and were ultimatelyclassified as not a border due to the lower local threshold value inparticular region. Other times, these potential cell borders weredisconnected and did not fully enclose a cell in the probability output.Then, during the post-processing pipeline, even if the disconnectedborder was classified as a border (when the image was binarized), itwould be removed by the area opening or the pruning morphologicaloperation, thus merging two or more cells into one big under-segmentedcell. Other times, these discrepancies occurred because U-Net was tryingto segment cells in a poor quality region of an image segmented a bigger“cell” that may not have been present. Less common, but still prevalent,were over-segmented cell discrepancies, when one cell was split into twoor more cells. Usually, this was due to a dark curve segment inside thecell that could have been mistaken for a cell border. Since thisoccurred less frequently, the under-segmented cells were primarilyaddressed within the GUI.

Thus, two methods were implemented to filter for theseunder-segmentation cases and highlight such cells for a second review.Briefly again, the first condition addressed cells with areas that weregreater than 3 standard deviations away from mean cell area within agiven image. The second condition addressed cells where, if the localthreshold was altered within the cell's interior, more cell bordersbecame apparent, splitting the once large cell into two or more realcells. These two methods were able to highlight a total of 167 cellsacross the 30 held-out test images. FIG. 24, discussed above, shows anexample of an EC image with cells that were highlighted for a secondrevision. Within the red circles of the probability output in FIG. 24,there are multiple disconnected border pixels that were eventuallyclassified as cell pixels as shown in the final automatic segmentationimage 2420. However, the method of altering the local threshold within asingle cell's interior resulted in the identification of these borders,thus they have been highlighted in the third image 2430. Using the pentool in the GUI software, borders were manually draw in and can be seenin the red circles of the fourth image 2440. Such edits would only takean extra 1 to 2 minutes after loading the images into the GUI software.

5. Discussion

U-Net proved to be a sufficient learning system with regards toautomatic segmentation of cell borders in clinical quality EC imagespost-keratoplasty. Quantitatively, the automatic segmentation algorithm,including pre-processing, the U-Net neural network, and post-processingpipeline was able to produce EC segmentations with Dice coefficient of0.87. However, there were over 500 newly identified cells in theheld-out test set that could not be quantitatively analyzed becausethere weren't ground truth cells to compare against. Of the newlyidentified cells, only 9% would require manual editing. Furthermore,there were a couple cells (less than 3%) within the ground truth regionthat were either over- or under-segmented by the algorithm. Thus, afterperforming the visual analysis study, it was evident a couple manualedits were often involved for the images to be an accuraterepresentation of the cell borders within the images.

The second example use case developed a semi-automated segmentation GUIsoftware allowing users to compare information from the original ECimage, enhanced EC image, probability output, and segmentation borderoverlay on the enhanced EC image to make an informed decision aboutwhere to manually edit the final automatic segmentation. The GUIsoftware highlights cells based on their area with respect to theaverage cell area within the image. It also identifies cells that havebeen under-segmented, where based on the probability output from U-Net,border pixels may not have been classified because the local thresholdvalue in the cell region was too low. Manual editing following automaticsegmentation could be done in 3-5 minutes or less, which is much less incomparison to the 15-30 minutes now required for a full EC image manualsegmentation.

In summary, automated segmentation of cell borders using dep learning issuccessful even in challenging post-transplant EC images. While ECautomatic segmentation accuracy is very good, a couple small manualedits to each image could improve the accuracy even more. An interactivesemi-automated segmentation GUI software allows informative viewing andeasy editing of these EC images, assisting in potential future work ofcollecting accurate clinical morphometric calculations when helpingphysicians assess the healthiness of corneal transplants.

Additional Embodiments

In various example embodiments, method(s) discussed herein can beimplemented as computer executable instructions. Thus, in variousembodiments, a computer-readable storage device can store computerexecutable instructions that, when executed by a machine (e.g.,computer, processor), cause the machine to perform methods or operationsdescribed or claimed herein including operation(s) described inconnection with methods 100, 200, 300, 400, or any other methods oroperations described herein. While executable instructions associatedwith the listed methods are described as being stored on acomputer-readable storage device, it is to be appreciated thatexecutable instructions associated with other example methods oroperations described or claimed herein can also be stored on acomputer-readable storage device. In different embodiments, the examplemethods or operations described herein can be triggered in differentways. In one embodiment, a method or operation can be triggered manuallyby a user. In another example, a method or operation can be triggeredautomatically.

Embodiments discussed herein relate to training and/or employing deeplearning model(s) to automatically and/or semi-automatically segmentendothelial cells based at least in part on features and/or mappingsthat are not perceivable by the human eye, and involve computation thatcannot be practically performed in the human mind. As one example,machine learning and/or deep learning classifiers as described hereincannot be implemented in the human mind or with pencil and paper.Embodiments thus perform actions, steps, processes, or other actionsthat are not practically performed in the human mind, at least becausethey require a processor or circuitry to access digitized images storedin a computer memory and to extract or compute features that are basedon the digitized images and not on properties of tissue or the imagesthat are perceivable by the human eye. Embodiments described herein canuse a combined order of specific rules, elements, operations, orcomponents that render information into a specific format that can thenbe used and applied to create desired results more accurately, moreconsistently, and with greater reliability than existing approaches,thereby producing the technical effect of improving the performance ofthe machine, computer, or system with which embodiments are implemented.

Referring to FIG. 29, illustrated is a diagram of an example apparatus2900 that can facilitate training and/or employing deep learningmodel(s) to automatically and/or semi-automatically segment endothelialcells, according to various embodiments discussed herein. Apparatus 2800can be configured to perform various techniques discussed herein, forexample, various operations discussed in connection with sets ofoperations 100, 200, 300, and/or 400. Apparatus 2900 can comprise one ormore processors 2910 and memory 2920. Processor(s) 2910 can, in variousembodiments, comprise circuitry such as, but not limited to, one or moresingle-core or multi-core processors. Processor(s) 2910 can include anycombination of general-purpose processors and dedicated processors(e.g., graphics processors, application processors, etc.). Theprocessor(s) can be coupled with and/or can comprise memory (e.g., ofmemory 2920) or storage and can be configured to execute instructionsstored in the memory 2920 or storage to enable various apparatus,applications, or operating systems to perform operations and/or methodsdiscussed herein. Memory 2920 can be configured to store one or moredigitized microscopic images (e.g., obtained via specular or confocalmicroscopy, etc.) of corneal endothelial cells (e.g., for trainingand/or segmenting). Each of the image(s) can comprise a plurality ofpixels or voxels, each pixel or voxel having an associated intensity.Memory 2920 can be further configured to store additional data involvedin performing operations discussed herein, such as for determining aprognosis following keratoplasty (e.g., no adverse events vs. failure,no adverse events vs. non-rejection failure vs. rejection, etc.) from anoptical microscopic image and/or training a ML or DL model to generate aprognosis for keratoplasty from an optical microscopic image, asdiscussed in greater detail herein.

Apparatus 2900 can also comprise an input/output (I/O) interface 2930(e.g., associated with one or more I/O devices), a set of circuits 2950,and an interface 2940 that connects the processor(s) 2910, the memory2920, the I/O interface 2930, and the set of circuits 2950. I/Ointerface 2930 can be configured to transfer data between memory 2920,processor 2910, circuits 2950, and external devices, for example, amedical imaging device (e.g., specular or confocal microscope, etc.),and/or one or more remote devices for receiving inputs and/or providingoutputs to a clinician, patient, etc., such as optional personalizedmedicine device 2960.

The processor(s) 2910 and/or one or more circuits of the set of circuits2950 can perform one or more acts associated with a method or set ofoperations discussed herein, such as set of operations 100, 200, 300,and/or 400. In various embodiments, different acts (e.g., differentoperations of a set of operations) can be performed by the same ordifferent processor(s) 2910 and/or one or more circuits of the set ofcircuits 2950.

Apparatus 2900 can optionally further comprise personalized medicinedevice 2960. Apparatus 2900 can be configured to provide the segmentedendothelial cells for the patient, and/or other data (e.g., keratoplastyprognosis, suggested interventions, etc.) to personalized medicinedevice 2960. Personalized medicine device 2960 may be, for example, acomputer assisted diagnosis (CADx) system or other type of personalizedmedicine device that can be used to facilitate monitoring and/ortreatment of an associated medical condition. In some embodiments,processor(s) 2910 and/or one or more circuits of the set of circuits2950 can be further configured to control personalized medicine device2960 to display the segmented endothelial cells for the patient or otherdata on a computer monitor, a smartphone display, a tablet display, orother displays.

Examples herein can include subject matter such as an apparatus, amicroscope (e.g., specular, confocal, etc.), a personalized medicinesystem, a CADx system, a processor, a system, circuitry, a method, meansfor performing acts, steps, or blocks of the method, at least onemachine-readable medium including executable instructions that, whenperformed by a machine (e.g., a processor with memory, anapplication-specific integrated circuit (ASIC), a field programmablegate array (FPGA), or the like) cause the machine to perform acts of themethod or of an apparatus or system for segmenting endothelial cellsand/or generating a prognosis for keratoplasty, according to embodimentsand examples described.

Example 1 is a non-transitory computer-readable medium storingcomputer-executable instructions that, when executed, cause a processorto perform operations, comprising: accessing an optical microscopy imagecomprising a set of corneal endothelial cells of a patient of akeratoplasty; pre-processing the optical microscopy image to generate apre-processed optical microscopy image via correcting for at least oneof shading or illumination artifacts in the optical microscopy image;segmenting, based at least in part on a trained deep learning (DL)model, a plurality of corneal endothelial cells of the set of cornealendothelial cells in the pre-processed optical microscopy image; anddisplaying, via a graphical user interface (GUI), at least the segmentedplurality of corneal endothelial cells.

Example 2 comprises the subject matter of any variation of any ofexample(s) 1, wherein the operations further comprise analyzing thesegmented plurality of corneal endothelial cells to determine whetherone or more cells of the segmented plurality of corneal endothelialcells are potentially under-segmented or potentially over-segmented.

Example 3 comprises the subject matter of any variation of any ofexample(s) 2, wherein one or more cells of the segmented plurality ofcorneal endothelial cells are potentially under-segmented or potentiallyover-segmented, and wherein the operations further comprise identifying,on the GUI, the one or more cells of the segmented plurality of cornealendothelial cells that are potentially under-segmented or potentiallyover-segmented.

Example 4 comprises the subject matter of any variation of any ofexample(s) 3, wherein the operations further comprise modifying, basedon a user input, a segmentation of at least one of the one or more cellsof the segmented plurality of corneal endothelial cells that arepotentially under-segmented or potentially over-segmented.

Example 5 comprises the subject matter of any variation of any ofexample(s) 3-4, wherein the operations further comprise determining thata first cell of the one or more cells is potentially under-segmentedbased on the first cell having an area greater than a first threshold.

Example 6 comprises the subject matter of any variation of any ofexample(s) 5, wherein the first threshold is three standard deviationsgreater than a mean cell area of the segmented plurality of cornealendothelial cells.

Example 7 comprises the subject matter of any variation of any ofexample(s) 3-6, wherein the optical microscopy image comprises aplurality of pixels, wherein segmenting the plurality of cornealendothelial cells is based at least in part on a binarization thresholdfor identifying whether each pixel of the plurality of pixelscorresponds to a cell interior or a cell boundary, and wherein theoperations further comprise determining that a first cell of the one ormore cells is potentially under-segmented based on the first cellcorresponding to two or more cells when the segmentation threshold ischanged.

Example 8 comprises the subject matter of any variation of any ofexample(s) 3-7, wherein the operations further comprise determining thata first cell of the one or more cells is potentially over-segmentedbased on the first cell having an area lower than a first threshold.

Example 9 comprises the subject matter of any variation of any ofexample(s) 8, wherein the first threshold is three standard deviationslower than a mean cell area of the segmented plurality of cornealendothelial cells.

Example 10 comprises the subject matter of any variation of any ofexample(s) 1-9, wherein the DL model is a convolutional neural network(CNN).

Example 11 comprises the subject matter of any variation of any ofexample(s) 10, wherein the CNN has a U-Net architecture.

Example 12 comprises the subject matter of any variation of any ofexample(s) 1, wherein the segmenting comprises performing binarizing aprobability map generated by the DL model based on one of an adaptivethreshold or an Otsu threshold.

Example 13 is a non-transitory computer-readable medium storingcomputer-executable instructions that, when executed, cause a processorto perform operations, comprising: accessing a training set comprising aplurality of optical microscopy images, wherein each optical microscopyimage of the training set comprises an associated set of cornealendothelial cells of a patient associated with that optical microscopyimage, and wherein each optical microscopy image of the training set isassociated with a ground truth segmentation of plurality of cornealendothelial cells of the set of corneal endothelial cells of thatoptical microscopy image; pre-process each optical microscopy image ofthe training set to correct for at least one of shading or illuminationartifacts; and based at least on each pre-processed optical microscopyimage and the ground truth segmentation for each optical microscopyimage, training a deep learning model to segment a plurality of cornealendothelial cells of a set of corneal endothelial cells of an additionalpre-processed optical microscopy image.

Example 14 comprises the subject matter of any variation of any ofexample(s) 13, wherein the deep learning model is a convolutional neuralnetwork (CNN).

Example 15 comprises the subject matter of any variation of any ofexample(s) 14, wherein the CNN has a U-Net architecture.

Example 16 is an apparatus, comprising: memory configured to store anoptical microscopy image comprising a set of corneal endothelial cellsof a patient of a keratoplasty; one or more processors configured toperform operations comprising: pre-processing the optical microscopyimage to generate a pre-processed optical microscopy image viacorrecting for at least one of shading or illumination artifacts in theoptical microscopy image; segmenting, based at least in part on atrained deep learning (DL) model, a plurality of corneal endothelialcells of the set of corneal endothelial cells in the pre-processedoptical microscopy image; and displaying, via a graphical user interface(GUI), at least the segmented plurality of corneal endothelial cells.

Example 17 comprises the subject matter of any variation of any ofexample(s) 16, wherein the operations further comprise analyzing thesegmented plurality of corneal endothelial cells to determine whetherone or more cells of the segmented plurality of corneal endothelialcells are potentially under-segmented or potentially over-segmented.

Example 18 comprises the subject matter of any variation of any ofexample(s) 17, wherein one or more cells of the segmented plurality ofcorneal endothelial cells are potentially under-segmented or potentiallyover-segmented, and wherein the operations further comprise identifying,on the GUI, the one or more cells of the segmented plurality of cornealendothelial cells that are potentially under-segmented or potentiallyover-segmented.

Example 19 comprises the subject matter of any variation of any ofexample(s) 18, wherein the operations further comprise modifying, basedon a user input, a segmentation of at least one of the one or more cellsof the segmented plurality of corneal endothelial cells that arepotentially under-segmented or potentially over-segmented.

Example 20 comprises the subject matter of any variation of any ofexample(s) 18-19, wherein the operations further comprise determiningthat a first cell of the one or more cells is potentiallyunder-segmented based on the first cell having an area greater than afirst threshold.

Example 21 comprises the subject matter of any variation of any ofexample(s) 20, wherein the first threshold is three standard deviationsgreater than a mean cell area of the segmented plurality of cornealendothelial cells.

Example 22 comprises the subject matter of any variation of any ofexample(s) 18-20, wherein the optical microscopy image comprises aplurality of pixels, wherein segmenting the plurality of cornealendothelial cells is based at least in part on a binarization thresholdfor identifying whether each pixel of the plurality of pixelscorresponds to a cell interior or a cell boundary, and wherein theoperations further comprise determining that a first cell of the one ormore cells is potentially under-segmented based on the first cellcorresponding to two or more cells when the segmentation threshold ischanged.

Example 23 comprises the subject matter of any variation of any ofexample(s) 18-22, wherein the operations further comprise determiningthat a first cell of the one or more cells is potentially over-segmentedbased on the first cell having an area lower than a first threshold.

Example 24 comprises the subject matter of any variation of any ofexample(s) 23, wherein the first threshold is three standard deviationslower than a mean cell area of the segmented plurality of cornealendothelial cells.

Example 25 comprises an apparatus comprising means for executing any ofthe described operations of examples 1-24.

Example 26 comprises a machine readable medium that stores instructionsfor execution by a processor to perform any of the described operationsof examples 1-24.

Example 27 comprises an apparatus comprising: a memory; and one or moreprocessors configured to: perform any of the described operations ofexamples 1-24.

References to “one embodiment”, “an embodiment”, “one example”, and “anexample” indicate that the embodiment(s) or example(s) so described mayinclude a particular feature, structure, characteristic, property,element, or limitation, but that not every embodiment or examplenecessarily includes that particular feature, structure, characteristic,property, element or limitation. Furthermore, repeated use of the phrase“in one embodiment” does not necessarily refer to the same embodiment,though it may.

“Computer-readable storage device”, as used herein, refers to a devicethat stores instructions or data. “Computer-readable storage device”does not refer to propagated signals. A computer-readable storage devicemay take forms, including, but not limited to, non-volatile media, andvolatile media. Non-volatile media may include, for example, opticaldisks, magnetic disks, tapes, and other media. Volatile media mayinclude, for example, semiconductor memories, dynamic memory, and othermedia. Common forms of a computer-readable storage device may include,but are not limited to, a floppy disk, a flexible disk, a hard disk, amagnetic tape, other magnetic medium, an application specific integratedcircuit (ASIC), a compact disk (CD), other optical medium, a randomaccess memory (RAM), a read only memory (ROM), a memory chip or card, amemory stick, and other media from which a computer, a processor orother electronic device can read.

“Circuit”, as used herein, includes but is not limited to hardware,firmware, software in execution on a machine, or combinations of each toperform a function(s) or an action(s), or to cause a function or actionfrom another logic, method, or system. A circuit may include a softwarecontrolled microprocessor, a discrete logic (e.g., ASIC), an analogcircuit, a digital circuit, a programmed logic device, a memory devicecontaining instructions, and other physical devices. A circuit mayinclude one or more gates, combinations of gates, or other circuitcomponents. Where multiple logical circuits are described, it may bepossible to incorporate the multiple logical circuits into one physicalcircuit. Similarly, where a single logical circuit is described, it maybe possible to distribute that single logical circuit between multiplephysical circuits.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim.

Throughout this specification and the claims that follow, unless thecontext requires otherwise, the words ‘comprise’ and ‘include’ andvariations such as ‘comprising’ and ‘including’ will be understood to beterms of inclusion and not exclusion. For example, when such terms areused to refer to a stated integer or group of integers, such terms donot imply the exclusion of any other integer or group of integers.

To the extent that the term “or” is employed in the detailed descriptionor claims (e.g., A or B) it is intended to mean “A or B or both”. Whenthe applicants intend to indicate “only A or B but not both” then theterm “only A or B but not both” will be employed. Thus, use of the term“or” herein is the inclusive, and not the exclusive use. See, Bryan A.Garner, A Dictionary of Modern Legal Usage 624 (2d. Ed. 1995).

While example systems, methods, and other embodiments have beenillustrated by describing examples, and while the examples have beendescribed in considerable detail, it is not the intention of theapplicants to restrict or in any way limit the scope of the appendedclaims to such detail. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing the systems, methods, and other embodiments described herein.Therefore, the invention is not limited to the specific details, therepresentative apparatus, and illustrative examples shown and described.Thus, this application is intended to embrace alterations,modifications, and variations that fall within the scope of the appendedclaims.

What is claimed is:
 1. A non-transitory computer-readable medium storingcomputer-executable instructions that, when executed, cause a processorto perform operations, comprising: accessing an optical microscopy imagecomprising a set of corneal endothelial cells of a patient of akeratoplasty; pre-processing the optical microscopy image to generate apre-processed optical microscopy image via correcting for at least oneof shading or illumination artifacts in the optical microscopy image;segmenting, based at least in part on a trained deep learning (DL)model, a plurality of corneal endothelial cells of the set of cornealendothelial cells in the pre-processed optical microscopy image; anddisplaying, via a graphical user interface (GUI), at least the segmentedplurality of corneal endothelial cells.
 2. The non-transitorymachine-readable medium of claim 1, wherein the operations furthercomprise analyzing the segmented plurality of corneal endothelial cellsto determine whether one or more cells of the segmented plurality ofcorneal endothelial cells are potentially under-segmented or potentiallyover-segmented.
 3. The non-transitory machine-readable medium of claim2, wherein one or more cells of the segmented plurality of cornealendothelial cells are potentially under-segmented or potentiallyover-segmented, and wherein the operations further comprise identifying,on the GUI, the one or more cells of the segmented plurality of cornealendothelial cells that are potentially under-segmented or potentiallyover-segmented.
 4. The non-transitory machine-readable medium of claim3, wherein the operations further comprise modifying, based on a userinput, a segmentation of at least one of the one or more cells of thesegmented plurality of corneal endothelial cells that are potentiallyunder-segmented or potentially over-segmented.
 5. The non-transitorymachine-readable medium of claim 3, wherein the operations furthercomprise determining that a first cell of the one or more cells ispotentially under-segmented based on the first cell having an areagreater than a first threshold.
 6. The non-transitory machine-readablemedium of claim 5, wherein the first threshold is three standarddeviations greater than a mean cell area of the segmented plurality ofcorneal endothelial cells.
 7. The non-transitory machine-readable mediumof claim 3, wherein the optical microscopy image comprises a pluralityof pixels, wherein segmenting the plurality of corneal endothelial cellsis based at least in part on a binarization threshold for identifyingwhether each pixel of the plurality of pixels corresponds to a cellinterior or a cell boundary, and wherein the operations further comprisedetermining that a first cell of the one or more cells is potentiallyunder-segmented based on the first cell corresponding to two or morecells when the segmentation threshold is changed.
 8. The non-transitorymachine-readable medium of claim 3, wherein the operations furthercomprise determining that a first cell of the one or more cells ispotentially over-segmented based on the first cell having an area lowerthan a first threshold.
 9. The non-transitory machine-readable medium ofclaim 8, wherein the first threshold is three standard deviations lowerthan a mean cell area of the segmented plurality of corneal endothelialcells.
 10. The non-transitory computer-readable medium of claim 1,wherein the DL model is a convolutional neural network (CNN).
 11. Thenon-transitory computer-readable medium of claim 10, wherein the CNN hasa U-Net architecture.
 12. The non-transitory computer-readable medium ofclaim 1, wherein the segmenting comprises performing binarizing aprobability map generated by the DL model based on one of an adaptivethreshold or an Otsu threshold.
 13. A non-transitory computer-readablemedium storing computer-executable instructions that, when executed,cause a processor to perform operations, comprising: accessing atraining set comprising a plurality of optical microscopy images,wherein each optical microscopy image of the training set comprises anassociated set of corneal endothelial cells of a patient associated withthat optical microscopy image, and wherein each optical microscopy imageof the training set is associated with a ground truth segmentation ofplurality of corneal endothelial cells of the set of corneal endothelialcells of that optical microscopy image; pre-processing each opticalmicroscopy image of the training set to correct for at least one ofshading or illumination artifacts; and based at least on eachpre-processed optical microscopy image and the ground truth segmentationfor each optical microscopy image, training a deep learning model tosegment a plurality of corneal endothelial cells of a set of cornealendothelial cells of an additional pre-processed optical microscopyimage.
 14. The non-transitory computer-readable medium of claim 13,wherein the deep learning model is a convolutional neural network (CNN).15. The non-transitory computer-readable medium of claim 14, wherein theCNN has a U-Net architecture.
 16. An apparatus, comprising: memoryconfigured to store an optical microscopy image comprising a set ofcorneal endothelial cells of a patient of a keratoplasty; one or moreprocessors configured to perform operations comprising: pre-processingthe optical microscopy image to generate a pre-processed opticalmicroscopy image via correcting for at least one of shading orillumination artifacts in the optical microscopy image; segmenting,based at least in part on a trained deep learning (DL) model, aplurality of corneal endothelial cells of the set of corneal endothelialcells in the pre-processed optical microscopy image; and displaying, viaa graphical user interface (GUI), at least the segmented plurality ofcorneal endothelial cells.
 17. The apparatus of claim 16, wherein theoperations further comprise analyzing the segmented plurality of cornealendothelial cells to determine whether one or more cells of thesegmented plurality of corneal endothelial cells are potentiallyunder-segmented or potentially over-segmented.
 18. The apparatus ofclaim 17, wherein one or more cells of the segmented plurality ofcorneal endothelial cells are potentially under-segmented or potentiallyover-segmented, and wherein the operations further comprise identifying,on the GUI, the one or more cells of the segmented plurality of cornealendothelial cells that are potentially under-segmented or potentiallyover-segmented.
 19. The apparatus of claim 18, wherein the operationsfurther comprise modifying, based on a user input, a segmentation of atleast one of the one or more cells of the segmented plurality of cornealendothelial cells that are potentially under-segmented or potentiallyover-segmented.
 20. The apparatus of claim 18, wherein the operationsfurther comprise determining that a first cell of the one or more cellsis potentially under-segmented based on the first cell having an areagreater than a first threshold.
 21. The apparatus of claim 20, whereinthe first threshold is three standard deviations greater than a meancell area of the segmented plurality of corneal endothelial cells. 22.The apparatus of claim 18, wherein the optical microscopy imagecomprises a plurality of pixels, wherein segmenting the plurality ofcorneal endothelial cells is based at least in part on a binarizationthreshold for identifying whether each pixel of the plurality of pixelscorresponds to a cell interior or a cell boundary, and wherein theoperations further comprise determining that a first cell of the one ormore cells is potentially under-segmented based on the first cellcorresponding to two or more cells when the segmentation threshold ischanged.
 23. The apparatus of claim 18, wherein the operations furthercomprise determining that a first cell of the one or more cells ispotentially over-segmented based on the first cell having an area lowerthan a first threshold.
 24. The apparatus of claim 23, wherein the firstthreshold is three standard deviations lower than a mean cell area ofthe segmented plurality of corneal endothelial cells.